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(57) Abstract 

The invention is a method and apparatus to restore die 
obscured poriton of an image of a sequence of characters 
or other patterns in the presence of superimposed lines. 
The character restoration is accomplished t>y means of two 
kinds of statistical information, the frequency of occurrence 
of image pixel vahie obsenrations on either side of the 
obscuring line and the frequency of occurrence of character 
slice transitions within the width of the obscuring line. This 
statistical information is organized as a Hidden Markov 
Model. Having estimated the model's probabilities, the 
Viterbi algorithm is then employed to determine the optimal 
restoration across the length of the line. Two tasks are 
performed prior to the character restoration. First the Hidden 
Markov Model is trained. This task involves scanning across 
sequences of characters to estimate lx>th observation and 
tansition probabilities. Model construction is done off-line 
and typically only once. Prior to the run-time character 
restoration, the line itself is located using a modified version of 
the Hough transfonn. Local eigenfits are employed to reduce 
the necessary computation. 
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METHOD AND APPARATUS OF REMOVING SUPERIMPOSED 
LINES AND RESTORING PIXEL VALUES IN IMAGES 
CONTAINING RECURRING PATTERNS 

FIELD OF THE INVENTION 

The present invention relates generally to image processing and 
more particularly to removing lines superimposed on pixel-array images and* after 
5 line removal, restoring pixel values of the image by statistical means. The 
invention is applicable to images having recurring patterns including particularly 
textual characters. 

BACKGROUND OF THE D^fVENTION 

10 

Optical character recognition (OCR) and other means to 
automatically process images find wide application in the processing of labels, 
forms and other documents containing textual information. An impediment to the 
application of OCR to the reading of both package labels and forms is the presence 

1 5 of superimposed lines. Frequently the lines are preprinted as underlines on the 
form and the characters are typed or printed on the form with an incorrect 
registration, thereby causing the underlines to be superimposed at the base, or even 
through the middle^ of the characters. In other cases, spurious lines can be 
generated by ink marks, faulty portions of charge belts or laser primers, din, and 

20 the like. When the superimposed lines pass through or close to characters and 
other recurring patterns sought to be recognized, OCR algorithms, including those 
based on neural networks, frequendy fail. 

Many attempts have been made to solve this problem. A number of 
known techniques involve simply the removal of the superimposed lines from the 

25 image. A standard technique for identifying lines in images is to use algorithms 



based on the Hough transform. Usually, detecting a superimposed line iii an im^ge 
is fairly straightforward. However, in the presence of recuiring patterns in images, 
a number of problems can be encountered. In particular, methods of identifying 
superimposed lines in images containing characters many times offer too many 
false lines. Attempted resolution of all the false lines dramatically reduces the 
effectiveness of any method. The adaptive Hough technique, described by J. 
Dlingworth and J. Kittler in ''The Adaptive Hough Transform". IEEE PAMI-9, No, 
3, pp. 690-698, Sept. 1987, and incorporated herein by reference, attempts to 
reduce computational requirements by locating superimposed lines at multiple 
scales of parameter space. However, this and other modifications of Hough- 
transform methods have not been effective in the presence of characters. Many 
times the true superimposed line is not even detected in the presence of characters. 

Even where line identification and removal is successful, the 
removal of the lines generally will also remove a portion of the characters or other 
patterns sought to be recognized. The removal of the character portions frequentiy 
generates as many problems as existed before removal of the lines. These 
problems could be avoided if there were a way to restore the portions of the 
characters that are removed upon the removal of the superimposed line. 

There have been limited attempts to address the restoration of the 
character ponions that are removed upon the removal of a superimposed line. One 
such attempt is described by J. M. Gloger in "A Hough Transform Based 
Elimination of Auxiliary Lines in Addresses", United States Postal Service 
Advance Technology Conference, pp, A-85 - A-92, November 30-December 2. 
1992, which is incorporated herein by reference. In this method, line detection and 
character restoration involve working with the contours of the regions of interest 
Lines are detected with the Hough transform. Character restoration is 
accomplished by connecting the disconnected ends of character contours by the 
shortest combination of horizontal and vertical line segments. No contextual 
information along the sequence of characters is used. The present inventor knows 
of no attempts to use both boundary and context information for character or other 
pattern restoration. 

Howell et al., in U.S. Patent No. 5,226,091, have disclosed the use 
of finite state machines in general and hidden Markov models in particular to 
provide a method and apparatus to verify a signature based on position and velocity 
information obtained during the execution of a number of genuine signatures. 
However, the '091 patent contains no suggestion or teaching of how finite state 



machine techniques might be applied to the problem of restoring missing portions 
of characters or patterns in images. 

Thus a need exists to reliably and efficiently locate superimposed 
lines in images containing recurring patterns, such as characters, removing the 
superimposed lines, and replacing the removed lines with image segments that 
restore the proper shapes to the patterns. 

SUMMARY OF THE INVENTION 

The present invention solves the problems of the prior art described 
above. The invention provides an apparatus and method operating in two stages. 
In the first stage, superimposed lines in an image comprised of pixels are detected 
and removed from the image. The removal of the superimposed lines will also 
remove portions of patterns in the image where the superimposed lines intersected 
the patterns. In the second stage, the missing parts of the patterns arc restored by 
obtaining the values of valid pixels in a neighborhood of the missing parts of the 
patterns and perfomiing the restoration by using the values of the valid pixels and a 
doubly stochastic finite state machine to predict the most likely values for the 
missing parts of the patterns in the image. The finite state machine is trained prior 
to use in the restoration with the characters or other patterns sought to be restored. 

A finite state machine is a device capable of assuming a finite 
number of internal states, and which produces an output signal as a function of an 
input signal and its internal state. A finite state machine is further characterized by 
the fact that its internal state may change as a result of an input signal. In digital 
systems, the output signal and the input are typically vectors comprised of a 
number of discrete, digital values. A digital finite state nuichine is fully defined by 
specifying the set of allowable input vectors /, the set of allowable output vectors 
O, the set of internal states of the machine 5, and two functions: 

Next state function: Ij x 5^^ 5, for all Ij e / and S^^^S 
Output function: /y x £)■ for all 1^ e / and 5^^ e S 
where 5,- e S and O,- € O 

In a doubly stochastic finite state machine, the next state and the ouput are not 
uniquely determined, but occur with given probabilities. That is, for a given input 
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/: and internal state S^. each next sute 5/ may occur with one certain probability 
and each output may occur with another certain probability. 

In addressing the problem of character or other pattern restoration, 
5 two sources of information may be used. The first source is the boundary 
conditions on either side of the removed line. Any pattern restoration should fit in 
the observable portion of the patterns. The second source of restoration 
information is the context of the patterns themselves. Given that the restoration is 
limited to a sequence of recurring patterns, certain sub-sequences of pattern slices 
10 are more likely than others. The use of a finite stale machine enables the use of 
both of these sources of information. 

One embodiment of the invention is an apparatus for restoring the 
missing parts of recurring patterns in an image comprised of pixels, comprising an 
electronic camera for obtaining the values of valid pixels in a neighborhood of the 
1 5 missing parts of the recurring patterns, a storage device for storing data defining the 
states of a finite state machine model of the recurring patterns and previously trained 
with the recurring patterns, and a processor for using the values of valid pixels and 
at least some of the stored data to restore the values to pixels comprising the 
missing parts of the recurring patterns. 

Another embodiment is a method of line removal and image 
restoration in an image containing recurring patterns, where the image is comprised 
of pixels, comprising the steps of detecting lines to be removed at a resolution of 
one pixel with a generalized Hough transform, locating a cluster of the lines in 
Hough transform parameter space representing at least one detected line, 
determining the width and orientation of a thick line corresponding to the cluster, 
obtaining values of valid pixels in a neighborhood of the thick line, and restoring 
values to pixels comprising the thick lines using the values of valid pixels and at 
least a portion of stored data in a finite state machine previously trained with the 
recurring patterns. 

The invention may be used a single stage at a time. Thus one 
embodiment of the invention is a method of removing lines superimposed on an 
image comprised of a two-dimensional array of pixels, comprising the steps of 
detecting the lines at a resolution of one pixel with a generalized Hough transform, 
locating a cluster of the lines in Hough transform parameter space representing at 
least one detected line, and determining the width and orientation of a line 
corresponding to Uie cluster. Another embodiment of the invention is method of 
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restoring missing parts of recurring patterns in an image comprised of pixels, 
comprising the steps of obtaining values of valid pixels in a neighborhood of the 
missing parts of the recurring patterns, and restoring values to pixels comprising 
the missing parts of the recurring patterns using the values of valid pixels and at 
least a ponion of stored data in a finite state machine previously trained with the 
recurring patterns. The recurring patterns may be textual characters. The 
neighborhood of the missing parts of the recurring patterns can be a finite number 
of pixels along each of a series of scanning lines passing through the missing parts, 
the pixels being contiguous or not along each scanning line. The scanning lines 
may be perpendicular to the superimposed line. 

In a preferred embodiment, the invention comprises a charge- 
coupled device ("CCD") camera to acquire a pixel-array image of a label, form, or 
other scene containing recurring patterns. The image is stored or transferred to a 
memory device for processing, preferably by a general-purpose computer. The 
image may be thresholded so that the pixels in the image take on only one of two 
values, 0 (for black) and 1 (for white). In a typical thresholded image of an 
address label, the characters in the address will be made up of black (0) pixels 
against a white (1) background. The image will contain a number of contiguous 
black pixels for each character or other pattern to be recognized, with the number of 
pixels in each pattern depending on the size of the pattern and the resolution of the 
image. 

In the line removal stage of the invention, lines are first detected at 
the limit of resolution of the image (i.e., one pixel) with a modified version of the 
Hough transform. The work required to set up the Hough transform accumulator 
arrays is minimized by estimating image features in a local orientation. The thin 
lines detected by the Hough transform technique are grouped into clusters in line 
parameter space. From these line clusters, the parameters of the thick lines such as 
number of lines to be removed and the width and orientation of each are 
determined. All of the pixel values making up the thick lines to be removed are 
marked for subsequent processing. 

In a preferred embodiment, the neighborhood of the missing pans 
of the patterns is provided by scanning along each thick line to be removed. While 
scanning, a sequence of observation vectors are then mapped into the most 
probable character restoration using dynamic programming. A Hidden Markov 
Model ("HMM") comprises the finite state machine that underlies the character 
restoration. The model is trained from a sequence of character slices somewhat 
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wider than the thick line. During training, the state sequence is embetlded vHthin 
the observation sequence. Because of the embedding, model probabilities may be 
estimated by simple relative frequencies. 

In this preferred embodiment, two sequences of character slices are 
5 used. A training sequence is taken from characters without a superimposed line. 
Each of the elements of the sequ«ice of training vectors are observable. A run-time 
sequence is taken about the superimposed line. Elements of the run-time vectors 
located on the superimposed line are unobservable. 

The parameter values for the HMM are estimated from the training 
1 0 sequence. The general technique of training an HMM, Baum-Welch re-estimation, 
suffers from scaling problems. Because for training the actual states are embedded 
within the observations, a statistical method can be used. The present invention 
therefore uses simple relative frequencies to determine both the transition 
probabilities and the observation probabilities conditioned upon the states. 
15 Discussions of these techniques are contained in S. E. Levinson. L. R. Rabiner. 
M. M. Sondhi. "An Introduction to the Application of the Theory of Probabilistic 
Functions of a Markov Process to Automatic Speech Recognition". The Bell 
System Technical Journal. Vol. 62, No. 4, pp. 1035-1074, April. 1983. and L. R. 
Rabiner, "A Tutorial on Hidden Markov Models and Selected Applications in 
20 Speech Recognition", Proc. IEEE. Vol. 77, No. 2. pp. 257-285, Feb.. 1989, 
which are incorporated herein by reference. 

The Viterbi algorithm is employed to detemune the most probable 
sequence of state assignments. At each position slice across the superimposed line, 
the Viterbi algorithm takes into account two sources of information, the transition 
25 probabilities and the observation probabilities conditioned upon the states. Each 
run-time observation vector could correspond to any number of actual 
observations. The conditional probabUity of the run-time observation vector is then 
the sum of the probabilities of all possible observations. The Viterbi algorithm is 
discussed in G. D. Forney, Jr., "The Viterbi Algorithm", Proc. IEEE, Vol. 61. 
30 No. 3, pp. 268-278, March, 1973, which is incorporated herein by reference. 



It is thus an object of this invention to provide a method and an 
apparatus for quickly and reliably removing superimposed Unes from images. 

It is a further object of this invention to provide a method and an 
apparatus for restoring portions of patterns missing from images when 
superimposed lines are removed from the images. 
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It is a further object of this invention to provide a method and an 
apparatus which uses both the boundary conditions on either side of the 
superimposed line and the context of the patterns themselves to effect image 
restoration after removal of the superimposed line. 
5 It is a further object of this invention to automatically process 

images of labels with poor readability due to lines superimposed over characters in 
the labels so as to generate a high-quality image for OCR input 

The present invention meets these objects and overcomes the 
drawbacks of the prior art, as will be apparent from the detailed description of the 
10 embodiments that follow. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of the major components of a preferred 
embodiment of the invention. 

5 Figure 2 is a block diagram of the overall operational steps of a 

prefeired embodiment of the pattern restoration portion of the invention. 

Figure 3 is a block diagram of the overall operational steps of the 
line identification and removal portion of a prefeired embodiment of the invention. 

Figure 4 is a sequence of training observations in an example using 
a preferred embodiment of the invention. 

Figure 5 is a sequence of run-time observations in an example using 
a preferred embodiment of the invention. 

Figure 6 is the organization of finite stote machine models for the 
present invention. 

Figure 7 is an example of an input image for the invention. 
Figure 8 is an example of restored characters using the invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
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Figure 1 is a block diagram of the major components of the 
invention. A camera 11 captures an image of the scene to be processed. TTie image 
IS stored in an array of pixels, each pixel having a numeric representation of the 
intensity of light in the portion of the image represented by the pixel. Pixel values 
vary usually from 0 to 255 or 1023. with the maximum value corresponding to the 
brightest intensity (white) and with 0 or the minimum value corresponding to the 
least mtensity (black). Pixel arrays may be in any two-dimensional configuration, 
but typically the most straightforward to use is a rectangular array of rows and 
columns of pixels. 

In the preferred embodiment, camera 11 is a CCD camera, and the 
output from the CCD is stored in the memory of a general purpose computer 21. 
nie computer 21 is capable of being programmed to model a finite state machine, 
the finite state machine being trainable to recognize patterns of the type to be 
processed by the invention. The computer 21 is further capable of being 
programmed to cany out the foUowing detailed description of the operation of the 
35 invention. 
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Figure 2 is a block diagram of the overall operational steps of a 
preferred embodiment of the pattern restoration portion of the invention. Prior to 
the steps of restoring missing portions of characters in an image, a finite state 
machine is constmcted. As shown in block 1 of figure 2, the finite state machine is 
5 capable of representing pixel transitions in neighborhoods of the patterns occurring 
in the image. Again prior to the restoration steps, the ^ite state machine is trained 
with as complete a set as is available of the patterns whose restoration is sought. 
This step is depicted in block 2 of Figure 2. The constraction and training of the 
finite machine may be accomplished off-line, ix., independent of the restoration 

10 steps and genially only once for a particular pattern set. 

The on-line steps of character restoration are depicted on the right 
side of Figure 2. As shown in blocks 3-7 of Figure 2, pixel data is obtained for an 
image containing recurring patterns. Additionally, information is obtained 
regarding which pixels are considered invalid in the image. A pixel may be invalid 

15 because its value is missing or because imperfections, such as superimposed lines, 
are present in the image. Next, valid pixel data is obtained in a neighborhood of 
one or more invalid pixels. The valid pixel data and the previously trained finite 
state machine are used to predict values for the invalid pixels, and the predicted 
values are assigned to the invalid pixels. 

20 Using a Hidden Markov Model ("HMM") as a particular choice of 

finite state machine, pattern restoration is accomplished by means of two kinds of 
statistical information, the frequency of occurrence of image pixel value 
observations on either side of the obscuring line and the frequency of occurrence of 
pattern slice transitions within the width of the obscuring line. This statistical 

25 information is organized as a HMM. Having estimated the model's probabilities, 
the Viterbi algorithm is then employed to determine the optimal restoration across 
the length of the line. Prior to the pattern restoration, the HMM is trained. This 
task involves scanning across sequences of patterns to estimate both observation 
and transition probabilities. Model construction is done off-line and typically only 

30 once. 

Also prior to the run-time pattern restoration, the superimposed line 
itself is located using a modified version of the Hough transform. Figure 3 is a 
block diagram of the overall operational steps of the line identification and removal 
ponion of a preferred embodiment of the invention. First, lines are detected at the 
35 limit of resolution of the image. These thin lines are grouped together in Hough 
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transfonn parameter space, identifying the thickness of the detected lihes. Lbcal 
eigenfits in the analysis are employed to reduce the necessaiy computation. 

The line detection and removal operational steps will be discussed in 
greater detail first, foUowed by additional detail on pattern restoration for the 
5 particular case where the patterns are characters in the English language. The 
invention works equally weU for characters in other languages and for restoring 
minority missing portions of other recurring patterns such as bar codes, image 
codes, and the like. 

The superimposed lines are located by a Hough transform 
10 technique. Hough transforms, reviewed in J. Eling worth, J. Kittler, "A Survey of 

the Hough Transform", Computer Vision, Graphics, and Image Processing, Vol. 

44, pp. 87-116, 1988, which review is incorporated herein by reference, are a 

known technique in image processing, useful in panicular to detect Unes in images. 

In the present invention, lines are detected at the resolution of the image using a 
1 5 Hough transfonn with a polar parameterization: 

xcos0 + ysin0=p 

O^0<2;r (1) 

0^P<oo 

For the line detection operation, the image coordinate frame is 
20 translated to the center of the image. This allows an unambiguous line 
parameterization in terms of angle and radius. 

Sometimes line orientation may be determined locally. Local 
orientation estimates are used to reduce the work required to set up the Hough's 
accumulator array. Given a set of image points with a local coordinate frame 
25 located at the center of the point set, the estimation of local orientation may be posed 
in terms of the minimization of a cost function: 

' -' (2) 

X. y black 

a. = COS© 

P = sin0 



30 



The minimization of Eiocai is determined by the application of the 
method of Lagrange multipliers to the least square fit of a line passing through the 
local origin with a unit length constraint on the line parameters. Define: 
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•■■[;] 



(4) 



(5) 



10 



Setting the derivative of Ejocai equal to zero implies the necessity to 
solve an eigensystem. The line orientation, with the orientation specified by the 
angle of the line's normal with respect to the local origin, is then implied by the 
eigenvector associated with the smallest eigenvalue. 



K' y 

X, y black 



r . r 



(6) 



15 Lei 



EX-' 

\ ^ y 

jc, y black 



a b'\ 
c d\ 



= 1 



then the line orientation is: 



(7) 



20 



©local = atan^^j= -atan^^-^j 
Since a and c are always greater than or equal to zero. 

, ^ o + <• - V(a' -h f ~ - lac 4fr^ 



(8) 



(9) 
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To translate the output of the inverse tangent function, usually in the 
range of -7C /2 to 7i /2, into the full range of orientations, 0 to 2k, it is necessary to 
locate the coordinates of the intersection of the line with the normal from the image 
center. Performing this step presents no difficulties to those skilled in the art, 
involving only the solution of a pair of linear equations. 

The span of orientation bins of the accumulator array to be updated 
are proportional to the orientation estimate's uncertainty as indicated by the ratio of 
eigenvalues: 
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20 



©local - ^ 



0 < ©local 



(10) 



Since the above technique locales lines at the limit of resolution, the 
located lines are necessarily thin. Where the image contains thick lines instead of 
5 thin ones, points in parameter space, each representing a located line, form 
identifiable clusters. Let each thin line, denoted by a point in parameter space with 
an accumulation count greater than a threshold, be represented by 1 = [p ©]' . Then 
an algorithm to identify clusters of thin lines, expressed in pseudo-C code, is: 

^ 0 1 . Identify the first line as the first cluster center: q = V Q = I li ) 

2. For each additional line, 1 { 

If (nunD j > threshold) create a new cluster. 

J 



Else update the cluster where j = argmin {£)• } { 
= It the number of lines in cluster j. 



25 



The class of model used for character restoration depends upon the 
line width. For a thick line at any orientation, the line width is determined by: 

30 w = max^U,- - Xjf + (y,- - yjf + 1 (ii) 

In the above equation, the , y, coordinates denote the point of 
intersection of a cluster member and a line perpendicular to the mean line. Given a 
point on the mean line, the intersection point with an orthogonal line is found from 
35 the solution of a pair of linear equations, well known to those skilled in the art. 
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* . A sequence of observation vectors are taken along the mean line. 

Each observation vector is peq)endicular to the mean line. For any point Xq and yp 

on the line referred to by the cluster center, the mean line, [p ©]' , the coordinates 
of a point offset orthogonally from Xq and y^ by a signed distance are: 

5 

X ~ Xq - yicos© (12) 

y = yo - yisin© (i3) 

10 Once the superimposed lines are located, the black pixels 

representing the line must be set to white in the background regions of the image 
and left black if within the character portion of the image. Consider a sequence of 
character slices as a sequence of symbols. Some sort of random process generated 
the symbol sequence. In the presence of a line through the characters, some portion 

15 of the symbols become hidden. Each character slice is an observation. Associated 
with the hidden portion of each observation is the state of a random process. 
Character restoration is accomplished by determining the most probable sequence of 
hidden states. 

Character restoration is predicated upon a priori knowledge of the 

20 construction of character slice sequences. Such knowledge is embodied in 
probabilities estimated from character slice sequences taken from line-free regions. 

The line-free portions of the image provide the data necessary to 
train the Hidden Markov Model ("HMM"), Figure 4 is a portion of actual training 
data. For this case of a five pixel wide line, three addition?.! samples arc taken on 

25 either side of the line. The samples on either side of the line provide valid data in a 
neighborhood of the parts of the image that will be missing when the line is 
removed. Three pixels on either side of the five-pixcl-wide line is a preferred 
number. Potentially more accurate character restoration will result if more than 
three pixels are used, at a cost of additional storage space and processing time. 

30 Conversely, a smaller neighborhood will speed processing but potentially lower the 
quality of restoration. It is desired that when presented with a line through 
characters, the restoration algorithm reproduce the most probable pattern sequence 
within the confines of the line. The training observations are examples of likely 
pattern sequences with likely states embedded within each pattern. 

35 An HMM defines a doubly stochastic process. In the following 

discussion, it is assumed that there are N states and M observation symbols. N 
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equal 2^ and M equals 2"*, where w is the line width and m is the observation 
symbol width. M is always greater than N. The HMM may be presented by: 

A = (A, B, 71 ) (14) 

5 

A is a N by N matrix of state transition probabilities. The state 

transition probability is the probability of the next state, given the current state. 

Each row of A sums to one. 7C is a N dimensional vector of initial state 

probabilities. The elements of n sum to one. 
^0 Define O = {o^, o^, Oj) to be a sequence of observation 

symbols. For training, state examples are embedded within the observations. An 

extracted state sequence is Q = {<?;. ^2 9r)- 

The goal of training is to maximize the probability of the observation 

sequence conditioned upon the model parameters. For this problem calculation of 
15 simple relative frequencies may substitute for Baum- Welch reestimation, as pointed 

out by S. Y. Kung, Digital Neural Networks, Prentice-Hall, 1993, and which is 

incorporated herein by reference. 



20 



^ __ count.of_transitions_from_state_i_to,state.j H^'k 

tot^l number of transitinnc fmm ctate i ^ ^ 



total _number _of _ transitions _ from _sute_ i 

g _ coum^of_the_occuirence_of_symbol_j_when_in_state_i 
'•7 total _numbcr_ of _oocurrcnccs_of_symbols_ when _ in _slate_i 



statc_ i _ is_ the_ first_state 
otherwise 



(16) 
(17) 



25 For sake of simplicity the simulation assumes that the starting state 

is always entirely white; i.e., the start sutc has all ones. Labels usually have a 
white border making this not an unreasonable assumption. 

It is possible that some states and some observation symbols may 
not appear in the training data. To prevent the failure of the restoration algorithm 

30 upon the occiurence of an observation not encountered during training, the A and B 
matrices are modified. If a row of A is entirely zeros, the state for that row is never 
observed during training, then all of the transitions from that state are made equally 
likely. 

When in a particular state, an observation symbol can never occur in 
35 the state cannot be embedded within the symbol. Hence even without any training 
data, certain observation symbols have zero probability of occurring from certain 
states. For those possible observation symbols, a phantom training sequence 
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Precedes the actual training sequence. Those symbols that could occur, but may or 
may not actually be observed, are given an initial symbol occurrence count of one. 
The initial occurrence count for impossible symbols is set to zero. 

Character restoration after training uses the Viterbi algorithm. The 
S goal of restoration is to fmd a state sequence that maximizes the probability of the 
state sequence conditioned upon the observation sequence and the model. 
Assuming that all observation sequences are equally likely, equivalent results are 
obtained by maximizing the joint probability of the state sequeilce and the 
observation sequence. From the definition of joint probability: 



10 



m<5|A) = Piai^)P(0\Q.A) (18) 



In the presence of a line, a portion of run-time observations is 
shown in Figure 5. 

IS Plugging in the HMM gives the joint probability of a pardcular state 

sequence and of a particular observation sequence as: -"^ 

PiQA^) = K.jia,^ n\ ^o.) (19) 

20 In the above equation, Og ^ is the probability of a transition from 

state qt to state q^^i^ an element of the A matrix. The line obscures some portion of 
the observation symbol. Hence any number of complete observation symbols 

could correspond to the observation vector. The probability of a disjunction of 
symbols is found by computing the sum of the individual probabilities, b^ io^) is 

25 then computed by summing all entries of the B matrix on the qi row^that have a 
symbol matching the observation vector. Symbol positions obscured positions 
obscured by the line, the 2's of each observation vector, may match either a zero or 
a one. 

Over the length of the line, performing a sequence of additions is 
30 more numerically stable than performing a sequence of multiplications. So instead 
of maximizing the joint probability, it is preferable to minimize the negative 
logarithm of the joint probability. Any logarithmic base could be used. The natural 
log is convenient. The following cost function is obtained: ^ 
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The obvious way to minimize the cost function is to simply 
enumerate all possible cases. This implies the necessity to check 2*"^ sequences, 
5 which is impractical. A remedy is the utilization of dynamic programming. The 
Viterbi algorithm uses forward dynamic programming to reduce the number of 
sequences checked to (T x 2^), a much more practical amount 

The cost function may be written recursively. Define 

10 5^.i=-log/> (e.OIA)l„ptt,t-i 

Then the Viterbi algorithm may be summarized as: 

5i(i) = -(logTt, +log6,(ai)) 

1 . Initialization: 

H',(/) = 0 z = l,2 N 

5, 0') = min{8,.i(/) " log^i / } " ^ogb:(Of ) / = 1, 2 N 

y = 1, 2 N 

2 . Forward recursion: 

4^(7) = argmin{5,.,(y) - loga, . } / = 2, 3 T 

i 

25 3. Path backtracking: 

= / = 7--l,r-2 1 

In step three, q* refers to the optimal state assignment. The 
30 sequence of best state assignments as well as the minimum costs are maintained in a 
doubly linked list called a trellis. (A trellis is discussed in D. P. Bensekas. 
Dynamic Programming: Deterministic and Stochastic Models, Prentice-Hall, 1987, 
which is incorporated herein by reference.) Due to the principle of optimality, at 
each slice of the trellis only the costs of 2*" paths need be updated. 
35 Figure 6 shows the method of adjusting the HMM for different 

thickness lines and different site characters. Lines of an image may be of variable, 
but uniform width. Line width is determined by the clustering process. The 
calculated thickness of the line may then direct the restoration algorithm to use the 
appropriate model. A different training sequence is required for each model. 
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* « 

Generalizing further gives a line of nonuniform width. Such a situation could be 
handled by having the restoration algorithm switch models upon encountering a 
change of the line width. No additional training sequences should be necessary. 
For both levels of generalization the width of the trellis of restored states varies 
5 according to the line width. 

The HMM for the illustrated restoration is trained on characters 
having the same size and font as those with the superimposed line. The 
probabilities of the model can change for characters differing in either their size or 
their font. The HMM to be used for characters differing in either their size or their 
10 font can be generated by trained on the chanced size or shape characters. With 
sufficient training, it is possible to characterize the dependency of the model 
probabilities on character size and font type. 



15 A label image containing a superimposed line, shown in Figure 7, 

was processed by the invention. 

The line detection routine used a parameter space threshold of 700 
and a cluster threshold of 1.0. Over a 19 by 19 pixel window, eigenfits estimated 
the local line orientation. Thin lines were clustered into a thick line about 3.4 pixels 

20 wide. Two pixels were added to the line width to account for non-uniform 
digitalization. The line width was then rounded to five pixels. Three pixels on 
either side of the thick line were included for observations. This gave a total 
observation symbol size of eleven pixels. 

The probabilities of the HMM are estimated from a sequence of over 

25 11,000 training observations. Multiple slices taken across the address label 
characters are appended together to create the training sequence. 

The resulting output is shown in Figure 8. Although there exist a 
few errors, the restoration appears quite good. Rather noticeable is the complete 
restoration of the letter E of AVE, Also interesting are the restorations of characters 

30 having no examples in the training sequence, such as the letter V and the period. 
Those characters that remain joined together could probably be separated by 
recursive OCR dependent segmentation methods. The point of this exercise is to 
transform the address label into something that the OCR algorithms are more likely 
to be successful with, while using a minimum of computational effort. 

35 While this invention has been described in detail with panicular 

reference to preferred embodiments thereof, it will be understood that variations and 
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modifications can be niade to these embodiments without departing from the spirit 
and scope of the invention as described herein and as defined in the appended 
claims. 
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I'claim: 



1 . A method of restoring missing parts of recurring patterns in 
an image comprised of pixels, comprising the steps of 

5 obtaining values of valid pixels in a neighborhood of said missing 

parts of said recurring patterns; and 

restoring values to pixels comprising said missing parts of said 
recurring patterns using said values of valid pixels and at least a portion of stored 
data in a doubly stochastic finite state machine previously trained with said 
10 recurring patterns. 

2. The method of claim 1, wherein said recurring patterns are 
textual characters. 



15 3. The method of claim 1, wherein said neighborhood of said 

missing parts of said recurring patterns comprises a finite number of pixels along 
each of a series of scanning lines passing through said missing parts. 

4. The method of claim 3, wherein said finite number of pixels 
20 are contiguous along each scanning line. 

5 . The method of claim 1 , wherein said missing parts of said 
recurring patterns arc caused by the prior removal of a line superimposed on said 
image. 

25 

6. The method of claim 5, wherein said neighborhood of said 
missing parts of said recurring patterns comprises a finite number of pixels along 
each of a series of scanning lines passing through said missing parts. 

30 7. The method of claim 6, wherein said scanning lines are 

perpendicular to said superimposed line. 

8. A method of removing lines superimposed on an image 
comprised of a two-dimensional array of pixels, comprising the steps of 
35 detecting said lines at a resolution of one pixel with a generalized 

Hough transform; 
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locating a cluster of said lines in Hough transform pararheter sf)ace 
representing at least one detected line; and 

determining the width and orientation of a line corresponding to said 

cluster. 

5 

9. A method of line removal and image restoration in an image 
containing recurring patterns, said image comprised of pixels, comprising the steps 
of 

detecting lines to be removed at a resolution of one pixel with a 
10 generalized Hough transform; 

locating a cluster of said lines in Hough transform parameter space 
representing at least one detected line; 

determining the width and orientation of a thick line corresponding 

to said cluster; 

15 obtaining values of valid pixels in a neighborhood of said thick line; 

and 

restoring values to pixels comprising said thick lines using said 
values of valid pixels and at least a portion of stored data in a doubly stochastic 
finite state machine previously trained with said recurring patterns. 

20 

10. An apparatus for restoring missing parts of recurring 
patterns in an image comprised of pixels, comprising 

means for obtaining the values of valid pixels in a neighborhood of 
said missing pans of said recurring patterns; 
25 means for storing data defining the states of a doubly stochastic 

finite state machine model of said recurring patterns and previously trained with 
said recurring patterns; and 

means, including means for using said values of valid pixels and at 
least some of the stored data in said means for storing data, for restoring the values 
30 to pixels comprising said missing parts of said recurring patterns. 

11. An apparatus for removing lines superimposed on an image 
comprised of a two-dimensional array of pixels, comprising 

means for detecting said lines at a resolution of one pixel with a 
35 generalized Hough transform; 
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• • 

means for locating a cluster of said lines in Hough transform 
parameter space representing at least one detected line; and 

means for determining the width and orientation of a line 
corresponding to said cluster. 

5 

12. An apparatus for line removal and image restoration in an 
image containing recurring patterns, said image comprised of pixels, comprising 

means for detecting lines to be removed at a resolution of one pixel 
with a generalized Hough transform; 
10 means for locating a cluster of said lines in Hough transform 

parameter space representing at least one detected line; 

means for determining the width and orientation of a thick line 
corresponding to said cluster; 

means for obtaining values of valid pixels in a neighborhood of said 

1 5 thick line; and 

means for restoring values to pixels comprising said thick lines 
using said values of valid pixels and at least a portion of stored data in a doubly 
stochastic finite state machine previously trained with said recurring patterns. 
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Detect lines with Hough transform at 
the limit of resolution of the image 
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Transform image to local coordinate 
system for detected line 
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Group thin lines together in Hough 
transform parameter space 
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Determine parameters of thick lines 
from grouping of thin lines 
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Mark thick line pixels for removal 
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